Storage system, method of controlling storage system, and storage device

ABSTRACT

The present invention provides a storage system and a method of controlling the storage system, in which a second site rapidly resumes system process when a first site is damaged. The storage system comprises a first site including a first storage device, a second site including a second storage device, and a third site including a third storage device, and the method of controlling the storage system comprises a step of making a logical volume of the second storage device consistent with a logical volume of the first storage device by remotely copying only the differential data between the logical volume of the first storage device and the logical volume of the second storage device from a logical volume of the third storage device to the logical volume of the second storage device when the first site is damaged.

CROSS-REFERENCES TO RELATED APPLICATIONS

This is a continuation application of U.S. Ser. No. 11/898,945, filedSep. 18, 2007, which is a continuation application of U.S. Ser. No.11/526,598, filed Sep. 26, 2006 (now U.S. Pat. No. 7,287,132), which isa continuation application of U.S. Ser. No. 11/196,418, filed Aug. 4,2005 (now U.S. Pat. No. 7,185,152), which is a continuation applicationof Ser. No. 10/823,618, filed Apr. 24, 2004 (now U.S. Pat. No.7,114,044).

This application relates to and claims priority from Japanese PatentApplication No. 2003-309194, filed on Sep. 1, 2003, the entiredisclosure of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a storage system and a method ofcontrolling the same.

2. Description of the Related Art

Disaster recovery is important in an information-processing system. As atechnique for recovery from a disaster, the standard technique is knownto store and manage a copy of the data on a storage device at a primarysite by a storage device located at a remote site (hereinafter, thistechnique is referred to as ‘remote copy’). According to this technique,when the primary site is damaged, processes performed at the primarysite are continuously performed at the remote site by using data in thestorage device located at the remote site.

In the above-described method, in order to continue to perform theprocess that was being performed at the primary site at the remote sitewhen the primary site is damaged, it is necessary to perform a remotecopy from the primary site to the remote site in real time. However, theprimary site and the remote site are often separated distantly from eachother. Therefore, if the remote copy from the primary site to the remotesite is performed in real time, it takes a considerable amount of timefor data communication, and the processing performance of the primarysite is decreased. In an information-processing system required for highavailability (HA), the processing performance of the primary site shouldnot be decreased, and the process should be rapidly resumed at theremote site when the primary site is damaged.

SUMMARY OF THE INVENTION

The present invention is designed to solve the aforementioned problem,and it is an object of the present invention to provide a storage systemand a method of controlling the same.

To achieve this object, there is provided a method of controlling astorage system comprising a first storage device having a first storagevolume provided at a first site, a second storage device having a secondstorage volume provided at a second site, and a third storage devicehaving a third storage volume provided at a third site, the storagedevices being connected so as to communicate with each other, whereinthe method includes the steps of: storing a copy of data stored in thefirst storage volume in the second storage volume at a first time;writing the copy of data written in the first storage volume into thethird storage volume; storing, in the third storage device, a writehistory of the data written in the first storage volume as a firstdifferential management table after the first time; and allowing thethird storage device to make the contents of the data stored in thesecond storage volume consistent with the contents of the data stored inthe first storage volume using the first differential management tableand the third storage volume of the third storage device.

Here, in an information-processing system in which a first site is aprimary site, a second site is a remote site, and a third site is alocal site provided in the vicinity of the first site, it becomespossible to make the second storage volume consistent with the firststorage volume by remotely copying only the differential data betweenthe first storage volume and the second storage volume from the thirdstorage volume to the second storage volume when the first site isdamaged.

Therefore, it is possible to provide a storage system and a method ofcontrolling the storage system having the above effects.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a schematic configuration of aninformation-processing system according to the present embodiment;

FIG. 2 is a diagram illustrating a configuration of a disk array deviceaccording to the present embodiment;

FIG. 3 is a diagram illustrating a configuration of aninformation-processing apparatus according to the present embodiment;

FIG. 4 is a diagram illustrating a configuration of a channel controlunit in a storage device according to the present embodiment;

FIG. 5 is a diagram illustrating a table stored in a shared memory in astorage device according to the present embodiment;

FIG. 6 is a diagram illustrating a pair management table according tothe present embodiment;

FIG. 7 is a diagram illustrating a differential management tableaccording to the present embodiment;

FIG. 8 is a diagram illustrating a change in the state of each storagedevice when a normal operation is performed at the first site in theinformation-processing system according to the present embodiment;

FIG. 9 is a flow diagram explaining a processing flow in a firstinformation-processing apparatus when a normal operation is performed atthe first site in the information-processing system according to thepresent embodiment;

FIGS. 10A and 10B are flow charts explaining a processing flow in athird storage device when a normal operation is performed at the firstsite in the information-processing system according to the presentembodiment; and

FIGS. 11A and 11B are diagrams illustrating an operation in which alogical volume of a second storage device is made to be consistent witha logical volume of the first storage device in order to shift a secondsite into use as the normal system in the information-processing systemaccording to the present embodiment.

DESCRIPTION OF THE EMBODIMENT Configuration of Information-ProcessingSystem

FIG. 1 is a block diagram illustrating the entire configuration of aninformation-processing system including a storage system 100 accordingto the present embodiment.

The information-processing system comprises a storage device, or storagesystem 10 (hereinafter, referred to as ‘a first storage device’ or‘first storage system’) provided at a first site, aninformation-processing apparatus 11 (hereinafter, referred to as ‘afirst information-processing apparatus’) accessing the first storagedevice, a storage device, or storage system 20 (hereinafter, referred toas ‘a second storage device’ or ‘second storage system’) provided at asecond site, an information-processing apparatus 21 (hereinafter,referred to as ‘a second information-processing apparatus’) accessingthe second storage device, and a storage device, or storage system 30(hereinafter, ‘a third storage device’ or ‘third storage system’)provided at a third site. As a detailed example of each of the sites,this could be a computer facility operated by an organization, such as auniversity or corporation, or a data center in which an operation of aweb server on the Internet or an operation of ASP (Application ServiceProvider) is performed. The information-processing system is built fordisaster recovery in the event of an earthquake, fire, typhoon, flood,lightning, act of terrorism, and the like. This is described in detailin U.S. patent application Ser. No. 10/096,375. The U.S. patentapplication Ser. No. 10/096,375 is included herein by reference.

In this embodiment, the first site is the above-described primary site,the second site is the above-described remote site, and the third siteis the above-described local site.

The storage devices 10, 20, and 30 provided at the respective sites areconnected so as to communicate with each other via a first network 40.The first network 40 is, for example, the Gigabit Ethernet (a registeredtrademark), an asynchronous transfer mode (ATM), a public line, and thelike.

Each of the information-processing apparatuses 11 and 21 is a computercomprising a central processing unit (CPU) and a memory, such as apersonal computer (PC), a workstation, or a main frame. Each of theinformation-processing apparatuses 11 and 21 may be composed of aplurality of computers connected to each other. In each of theinformation-processing apparatuses 11 and 21, an operating system is inuse. Application software is operated through the operating system.

For example, the application software could provide a function of anautomatic teller system of a bank or a seat reservation system of anairplane. The information-processing apparatuses 11 and 21, and thestorage devices 10 and 20 in the first site and the second site areinterconnected by communication means (e.g., a communication line and anetwork), respectively. The communication means is, for example, a localarea network (LAN), a storage area network (SAN), an Internet SmallComputer System Interface (ISCSI), a fibre channel, an EnterpriseSystems Connection (ESCON) (a registered trademark), a Fibre Connection(FICON) (a registered trademark), and the like.

FIG. 2 shows the detailed configuration of a disk array device describedas one example of the first to third storage devices 10, 20, and 30.Further, the first to third storage devices 10, 20, and 30 may be, forexample, semiconductor storage devices, in addition to the disk arraydevices. The disk array device comprises a channel control unit 101, aremote communication interface 102, a disk control unit 103, a sharedmemory 104, a cache memory 105, a switching control unit 106, which iscomposed of, for example, a crossbar switch, for connecting the abovecomponents such that they can communicate with each other, a managementterminal 107, and a plurality of storages, or disk devices 108.

The cache memory 105 is mainly used to temporarily store datacommunicated between the channel control unit 101 and the disk controlunit 103. For example, if data input/output commands received from theinformation-processing apparatuses 11 and 21 by the channel control unit101 are write commands, the channel control unit 101 writes write datareceived from the information-processing apparatuses 11 and 21 into thecache memory 105. Further, the disk control unit 103 reads out the writedata from the cache memory 105 and writes it into the storage 108.

The disk control unit 103 reads out data input/output (I/O) requestswhich are written into the shared memory 104 by the channel control unit101, and writes or reads data into or from the storage 108 according tocommands (e.g., commands of SCSI (Small Computer System Interface)standards) set up in the data I/O requests. The disk control unit 103writes the data read out from the storage 108 into the cache memory 105.Further, the disk control unit 103 transmits notice of the completion ofdata write or data read to the channel control unit 101. The diskcontrol unit 103 may have a function for controlling the storages 108 ina RAID (Redundant Array of Inexpensive Disks) level (e.g., 0, 1, 5)defined in a so-called RAID manner.

The storage 108 is, for example, a hard disk device. The storage 108 maybe integrated with the disk array device or may be independentlyprovided therefrom. A storage area provided by the storage device 108 ateach site is managed using a logical volume 109, which is a volumelogically set up in the storage area, as a unit. Writing or reading datainto or from the storage 108 can be performed by designating anidentifier, which is given to the logical volume 109.

The management terminal 107 is a computer for maintaining and managingthe disk array device or the storage 108. The change of software orparameters carried out in the channel control unit 101 or the diskcontrol unit 103 is performed by the instructions from the managementterminal 107. The management terminal 107 may be built in the disk arraydevice or may be separately provided therefrom.

The remote communication interface 102 is a communication interface (achannel extender) for data communication with other storage devices 10,20, and 30, and data duplicated by a remote copy, which will bedescribed later, is transmitted via the remote communication interface102. The remote communication interface 102 converts an interface (e.g.,an interface, such as a Fibre Channel, an ESCON (a registeredtrademark), or a FICON (a registered trademark)) of the channel controlunit 101 into a communication method for the first network 40. It allowsdata transmission between the storage devices 10, 20, and 30.

Further, the disk array device may be, for example, a device functioningas a NAS (Network Attached Storage) that is configured to accept datainput/output requests according to file name designation from theinformation-processing apparatuses 11 and 21 by a protocol, such as anNFS (Network File System), in addition to a device having theabove-stated configuration.

The shared memory 104 is accessible by both the channel control unit 101and the remote communication interface 102, and the disk control unit103. The shared memory is used to deliver data input/output requestcommands and also stores management information for the storage devices10, 20, and 30 or the storage device 108. In this embodiment, aconsistency group management table 200, a pair management table 201, anda differential management table 202 are stored in the shared memory 104,as shown in FIG. 5.

FIG. 3 is a block diagram illustrating the configuration of theinformation-processing apparatuses 11 and 21.

The information-processing apparatuses 11 and 21 each comprise a CPU110, a memory 120, a port 130, a recording medium reading device 140, aninput device 150, and an output device 160.

The CPU 110 is in charge of the control of the entireinformation-processing apparatuses 11 and 21, and realizes variousfunctions by executing programs stored in the memory 120. The recordingmedium reading device 140 is a device for reading out programs or datarecorded on a recording medium 170. The read programs or data are storedin the memory 120. Thus, for example, it is possible to read a storagedevice management program 121 or an application program 122, which isrecorded on the recording medium 170, from the recording medium 170using the recording medium reading device 140 and to store it in thememory 120. As the recording medium 170, a flexible disk, a CD-ROM, asemiconductor memory, and the like may be used. The recording mediumreading device 140 may be built in the information-processingapparatuses 11 and 21 or may be separately provided therefrom. Forexample, the input device 150 is used for inputting data into theinformation-processing apparatuses 11 and 21 by an operator. As theinput device 150, for example, a keyboard, a mouse, and the like areused. The output device 160 is a device for outputting information tothe outside. As the output device 160, for example, a display, aprinter, and the like are used. The port 130 is a device forcommunicating with the storage devices 10, 20, and 30. Further, the port130 may be used for communication between the information-processingapparatuses 11 and 21. In this case, for example, the storage devicemanagement program 121 or the application program 122 may be receivedfrom the information-processing apparatuses 11 and 21 via the port 130and may be stored in the memory 120.

The storage device management program 121 is a program for performingthe copy management of the data stored in the logical volume 109 that isincluded in the storage 108. Various commands for performing the copymanagement are transmitted to the storage devices 10, 20, and 30 toallow the copy management operation to be performed.

The application program 122 is a program for making theinformation-processing apparatus 11 and 21 execute various functions.For example, there are a function for realizing the above-statedautomatic teller system of the bank, a function for realizing the seatreservation system of the airplane, and the like.

FIG. 4 is a block diagram showing the configuration of the channelcontrol unit 101.

The channel control unit 101 comprises a CPU 211, a cache memory 212, acontrol memory 213, a port 215, and a bus 216.

The CPU 211 is in charge of the control of the entire channel controlunit 101 and executes a control program 214 stored in the control memory213. The copy management of data according to the present embodiment isrealized as the control program 214 stored in the control memory 213 isexecuted. The cache memory 212 is a memory for temporarily storing data,commands, and the like, which are communicated with theinformation-processing apparatuses 11 and 21. The port 215 is acommunication interface for communicating with theinformation-processing apparatuses 11 and 21 or for communicating withother devices in the storage devices 10, 20, and 30. The bus 216connects these devices to each other.

[Pair Management Table]

A pair management table 201 is a copy management table of the logicalvolume 109 of each of the storage devices 10, 20, and 30. The pairmanagement table 201 comprises ‘type of pair’, ‘copy manner’, ‘copysource device’, ‘copy destination device’, ‘copy source volume’, ‘copydestination volume’, ‘paring status’, and ‘consistency group’ columns.

A pair refers to a combination of logical volumes 109 formed by twological volumes 109. Further, a case in which two logical volumes 109forming a pair are in the same storage devices 10, 20 and 30 is called a‘local pair’, and a case in which they are in different storage devices10, 20 and 30 is called a ‘remote pair’. Of the logical volumes 109forming a pair, one is a main logical volume, and the other is asecondary logical volume. It is possible to combine a plurality ofsecondary logical volumes with one main logical volume.

If the information-processing apparatuses 11 and 21 instruct the copysource storage devices 10, 20, and 30 to form remote pairs, each of thecopy source storage devices 10, 20, and 30 update the pair managementtable 202 of the shared memory 104 in each of the copy source storagedevices 10, 20, and 30. Thereafter, the copy source storage devices 10,20, and 30 instruct the copy destination storage devices 10, 20, and 30to form the remote pairs. The copy source storage devices 10, 20, and 30having received the instruction update the pair management tables 202 ofthe shared memories 104 in the copy destination storage devices 10, 20,and 30.

The column ‘type of pair’ in the pair management table 201 representswhether the pair is a local pair or a remote pair. The column ‘copymanner’ represents whether a remote copy manner is a synchronous manneror an asynchronous manner when the pair is the remote pair. Further, theremote copy and the manner thereof will be described later. The columns‘copy source device’ and ‘copy destination device’ represent the storagedevices 10, 20, and 30 of the copy source and the storage devices 10,20, and 30 of the copy destination when the pair is the remote pair. Thecolumn ‘copy source volume’ represents a main logical volume of thepair, and the column ‘copy destination volume’ represents a secondarylogical volume of the pair.

The column ‘state of pair’ represents the state of the pair. As thestate of the pair, there are ‘under pair’, ‘under split’, and ‘underresynchronization’.

In case of the ‘under pair’, data written into the main logical volumefrom the information-processing apparatuses 11 and 21 is reflected tothe secondary logical volume. Thus, it is possible to surely equalizethe contents stored in the main logical volume with the contents storedin the secondary logical volume by matching the main logical volume tothe secondary logical volume.

In case of the ‘under split’, even though data is written frominformation-processing apparatuses 11 and 21 into the main logicalvolume, the data is not reflected to the secondary logical volume. Thatis, the ‘under split’ state is a state in which the correspondencebetween the main logical volume and the secondary logical volume at the‘under pair’ is released. Thus, the identification between the contentsof the main logical volume and the contents of the secondary logicalvolume is not secured. However, since the contents of the secondarylogical volume are not updated in the ‘under split’ state, the data canbe backed up at this time.

The ‘under resynchronization’ is a state in the course of shifting fromthe ‘under split’ to the ‘under pair’. That is, in the ‘under split’,the data update performed on the main logical volume is being reflectedto the secondary logical volume. If the reflection is completed, thestate of the pair becomes the ‘under pair’. Further, shifting the pairfrom the state of the ‘under split’ to the state of the ‘under pair’ iscalled pair re-forming.

The formation of a pair, the split of a pair, and the re-formation of apair are performed by the operator inputting instructions from the inputdevice 150 to the information-processing apparatuses 11 and 21 in whichthe storage device management program 121 is executed. The instructionsinput by the operator are transmitted to the channel control units 101in the storage devices 10, 20, and 30. Then, the channel control unit101 executes the control program 214 to perform the formation of thepair or a change of the state of the pair according to the instructions.The channel control unit 101 performs the control of the logical volume109 according to the pairing status of the formed pair. For example, thechannel control unit 101 reflects the copy of the updated data of themain logical volume to the secondary logical volume with respect to thepair in the ‘under pair’ state.

The change of the pairing status by the channel control unit 101 issequentially performed on each pair. The reason is that, for example, ina state in which a pair is formed by combining a plurality of secondarylogical volumes with one main logical volume as described above, when aplurality of pairing conditions is simultaneously changed, managementfor the main logical volume is complicated.

Further, while the formation of a pair or the shift of a paring statusis initiated by the instructions received by information-processingapparatuses 11 and 21 from the operator, the shift of the paring statusmay be automatically performed, for example, when a predetermined timeelapses without instructions from the operator. In addition, theformation of a pair or the shift of a paring status may be performed byreceiving instructions from the information-processing apparatuses 11and 21 connected via the port 215.

In this embodiment, a pair is set up as shown in the pair managementtable of FIG. 6. That is, the logical volume 109 (hereinafter, referredto as ‘a first logical volume’) at the first site forms a local pair.Further, the secondary logical volume (hereinafter, referred to as ‘afirst secondary logical volume’) of the local pair of the first logicalvolume and the logical volume 109 (hereinafter, referred to as ‘a secondlogical volume’) at the second site form a remote pair. The remote pairof the first main logical volume and the logical volume 109(hereinafter, referred to as ‘a third logical volume’) at the third siteis always in the ‘under pair’ state, and the third logical volume isalways consistent with the first main logical volume by the remote copyin a synchronous manner, which will be described later.

Further, data backup from the first logical volume to the second logicalvolume is performed as follows. First, the first storage device 10shifts the local pair of the first logical volume to the ‘under split’by the instructions from the first information-processing apparatus 11.When split is completed, the first storage device 10 re-forms the remotepair of the first secondary logical volume and the second logical volumeaccording to the instructions from the first information-processingapparatus 11. Further, the first information-processing apparatus cancontinue to perform the process using a main logical volume(hereinafter, referred to as ‘a first main logical volume’) of the localpair of the first logical volume during the re-formation of the remotepair.

[Consistency Group]

The number of a consistency group (i.e., a pair group) to which the pairbelongs is written into the column ‘consistency group’ of the paringstatus management table 201. Here, a consistency group refers to a groupformed of pairs of a plurality of logical volumes 109, which iscontrolled so that the shift to the split state thereof issimultaneously made. That is, a change of the paring status with respectto the plurality of pairs is sequentially performed on each pair asdescribed above. However, a plurality of pairs belonging to the sameconsistency group is controlled such that the shift to the split statethereof is simultaneously performed (hereinafter, it is referred to assimultaneity of the shift to the split state).

For example, a case is considered in which data is written from theinformation-processing apparatuses 11 and 21 to the storage devices 10,20, and 30 while the storage devices 10, 20, and 30 sequentially changea plurality of pairs belonging to the same consistency group from the‘under pair’ to the ‘under split’. If the consistency group has not beenformed, the write data is not reflected to the secondary logical volumewhen the data writing is intended with respect to the main logicalvolume of the pair the shift to the split state of which is completed.On the contrary, if the data writing is intended with respect to themain logical volume of the pair the shift to the split state of which isnot completed, the write data is reflected to the secondary logicalvolume. However, when the consistency group has been formed, the writedata is not reflected to the secondary logical volume, regardless ofwhether the shift to the split state is completed. This is because thedata writing from the processing devices 11 and 21 to the storagedevices 10, 20, and 30 is called after the split for the pair of theconsistency group has been initiated (the release of correspondence isinitiated).

As described above, forming a consistency group with respect to aplurality of pairs is particularly effective when one data item isstored in the plurality of logical volumes 109, for example, when thewrite data is too large to be stored in one logical volume 109, and whenone file data item is controlled so that it is stored in the pluralityof logical volumes 109.

In addition, securing simultaneity for the shift of each pair to thesplit state using the consistency group is effective even when there isa data write request or a data read request from theinformation-processing apparatuses 11 and 21 to the secondary logicalvolume.

That is, the reason is that, when the consistency group has not beenformed, data writing or data reading can be performed with respect to asecondary logical volume of the pair the shift to the split state ofwhich is completed, but data writing or data reading is prohibited withrespect to a secondary logical volume of the pair the shift to the splitstate of which is not completed.

Further, a split for a pair of the consistency group may be performed bydesignating the start time thereof. The start time of the split isinstructed by a command transmitted from the information-processingapparatuses 11 and 21.

[Remote Copy]

When the above-stated remote pairs are formed in the logical volumes 109of the first to third storage devices 10, 20, and 30, the copy sourcestorage devices 10, 20, and 30 transmit data copied by the remote copyto the copy destination storage devices 10, 20, and 30. Thecorresponding relationship between the main logical volumes of the copysource storage devices 10, 20, and 30 and the secondary logical volumesof the copy destination storage devices 10, 20, and 30 is set up in theabove-stated pair management table 201. When the information-processingapparatuses 11 and 21 write data to the main logical volumes of the copysource storage devices 10, 20, and 30 while the paring status is the‘under pair’, the copy source storage devices 10, 20, and 30 transmitthe copy of the data to the copy destination storage devices 10, 20, and30 via the first network 40. Then, the copy destination storage devices10, 20, and 30 write the received data to the secondary logical volumes.While the pairing status is ‘under split’, the copy source storagedevices 10, 20, and 30 do not transmit the copy of the data to the copydestination storage devices 10, 20, and 30 even when theinformation-processing apparatuses 11 and 21 write data to the mainlogical volumes of the copy source storage devices 10, 20, and 30. Thecopy source storage devices 10, 20, and 30 transmit the copy of the datato the copy destination storage devices 10, 20, and 30 at the timingwhen the paring status is shifted to the ‘under resynchronization’, andthen the copy destination storage devices 10, 20, and 30 write thereceived data to the secondary logical volumes. That is, the remote copyis performed to make the contents of the main logical volumes of thecopy source storage devices 10, 20, and 30 consistent with the contentsof the secondary logical volumes of the copy destination storage devices10, 20, and 30, wherein the copy source storage devices 10, 20, and 30correspond to the copy destination storage devices 10, 20, and 30.

Further, the remote copy manner includes a synchronous manner and anasynchronous manner, which are determined in the ‘copy manner’ column ofthe differential management table 202. In the case of the synchronousmanner, if the information-processing apparatuses 11 and 21 request thecopy source storage devices 10, 20, and 30 to write data into the mainlogical volumes, the copy source storage devices 10, 20, and 30 firstwrite data into the main logical volumes thereof. In addition, the copysource storage devices 10, 20, and 30 transmit the same data as thewrite data to copy destination storage devices 10, 20, and 30. Then, thecopy destination storage devices 10, 20, and 30 write the received datainto the secondary logical volumes and notify that fact to the copysource storage devices 10, 20, and 30. The copy source storage devices10, 20, and 30 that have received this notification give notice thatdata writing has been completed to the information-processingapparatuses 11 and 21.

In such a synchronous manner, the information-processing apparatuses 11and 21 have notice of the completion of data writing after it isconfirmed that data is written to both the main logical volumes andsecondary logical volumes. Accordingly, in the synchronous manner, thecorrespondence between the contents of the main logical volumes and thecontents of the secondary logical volumes is secured at the time whenthe information-processing apparatuses 11 and 21 receive notice of thecompletion. However, the information-processing apparatuses 11 and 21receive notice of the completion until data is completely written intothe secondary logical volume. Thus, in the synchronous manner, responsetime until the completion notice returns to the information-processingapparatuses 11 and 21 after a data write request is transmitted frominformation-processing apparatuses 11 and 21 to the copy source storagedevices 10, 20, and 30 is generally longer than that in the asynchronousmanner.

Meanwhile, in the asynchronous manner, when the information-processingapparatuses 11 and 21 request the copy source storage devices 10, 20,and 30 to write data into the main logical volumes, the copy sourcestorage devices 10, 20, and 30 write data into the main logical volumes.Further, the copy source storage devices 10, 20, and 30 transmit thesame data as the write data to the copy destination storage devices 10,20, and 30. The copy destination storage devices 10, 20, and 30 writethe received data into the secondary logical volumes and notify thatfact to the copy source storage devices 10, 20, and 30. Here, if thecopy source storage devices 10, 20, and 30 write data to the mainlogical volumes, they immediately transmit the completion notice to theinformation-processing apparatuses 11 and 21, irregardless of whetherdata is written to the secondary logical volumes of the copy destinationstorage devices 10, 20, and 30. Thus, in the asynchronous manner,response time with respect to the information-processing apparatuses 11and 21 is generally shorter as compared to the synchronous manner.However, the data coincidence between the main logical volumes and thesecondary logical volumes is not necessarily secured. Further, in thecase of the asynchronous manner, the copy source storage devices 10, 20,and 30 manage data that is not reflected to the secondary logicalvolumes of the copy destination storage devices 10, 20, and 30.

In this embodiment, a remote pair between the first main logical volumeand the third logical volume performs a remote copy in the synchronousmanner. Thus, the contents of the first main logical volume are alwaysconsistent with the contents of the third logical volume.

[Differential Management Table]

The differential management table 202 shows whether any block of acertain logical volume 109 is updated on the basis of a predeterminedtime. As shown in FIG. 5, the table is generally recorded in the sharedmemories 104 of the storage devices 10, 20, and 30 in a bitmap format,and each bit corresponds to a block in the logical volume 109. Thedifferential management table 202 in which an initial state of each itemis ‘0’ is recorded in the shared memories 104 of the storage devices 10,20, and 30. If the storage devices 10, 20, and 30 write data in thelogical volumes 109, each of them updates a bit on the differentialmanagement table, which indicates a block in which the data is stored,to ‘1’.

In this embodiment, whether the third storage device 30 updates anyblock of the third logical volume on the basis of the time when thelocal pair of the first logical volume is split is recorded in theshared memory 104 of the third storage device 30 as the differentialmanagement table 202. As described above, the remote pair between thefirst main logical volume and the third logical volume is always the‘under pair’, thereby performing the remote copy in the synchronousmanner. Thus, the differential management table 202 of the third storagedevice 30 indicates a write history of data recorded in the first mainlogical volume after the local pair of the first logical volume issplit. That is, in the differential management table 202, data in thethird logical volume of a block in which ‘1’ is set up is data recordedin the first main logical volume after the local pair of the firstlogical volume is split. In addition, the first information-processingapparatus 11 backs up the first secondary logical volume to the secondlogical volume in a state in which the local pair of the first logicalvolume is split.

In this way, when the first site is damaged, the third storage device 30makes the second logical volume consistent with the first main logicalvolume by remotely copying only data in the third logical volume, whichis set up to ‘1’ in the differential management table 202 of the thirdstorage device 30, to the second logical volume.

However, when the first site is damaged while the first storage device10 performs a remote copy from the first secondary logical volume to thesecond logical volume, the second logical volume has not been updated soas to be consistent with the first main logical volume at the time whenthe local pair of the first logical volume is split the last time. Thus,the third storage device 30 can make the second logical volumeconsistent with the first main logical volume only with the differentialmanagement table 202 at the time when the first logical volume is splitthe last time.

Thus, in this embodiment, the third storage device 30 records thedifferential management table 202 at the time when the local pair of thefirst logical volume is split the time before last and the last time.Accordingly, when the first site is damaged while the first storagedevice 11 performs a remote copy from the first secondary logical volumeto the second logical volume, the third storage device 30 can make thesecond logical volume consistent with the first main logical volumeusing the differential management table 202 recorded at the time whenthe local pair of the first logical volume is split the time beforelast.

FIG. 7 shows an example in which the first site is damaged while thefirst storage device 10 is performing a remote copy from the firstsecondary logical volume to the second logical volume. In thedifferential management table 202, ‘1’ is set up with respect to a blockof the third logical volume on which the third storage device 30performs update, and FIG. 7 shows the contents of write data in thelogical volume for the sake of convenience in the present invention. Asshown in FIG. 7, at the time when it is damaged, update is completed upto the data ‘5’ in the remote copy from the first secondary logicalvolume to the second logical volume, which is performed by the firststorage device 10. However, since the update of the subsequent data ‘9’is not completed, the second logical volume is not consistent with thefirst main logical volume at the time when the local pair of the firstlogical volume is split the last time. Thus, the third storage device 30remotely copies, to the second logical volume, data ‘1’, ‘5’, ‘9’, ‘3’,‘4’, and ‘2’ of the blocks 1 to 6 in which ‘1’ is set up in thedifferential management table at the time when the local pair of thefirst logical volume is split the time before last. Accordingly, thethird storage device 30 can make the second logical volume consistentwith the first main logical volume at the time when the first site isdamaged.

Further, if the storage devices 10, 20, and 30 receive differentialdelete requests from the information-processing apparatuses 11 and 21,they update all bits of the differential management tables 202 to ‘0’.The information-processing apparatuses 11 and 21 may perform thedifferential delete requests by designating a differential delete starttime. In addition, the information-processing apparatuses 11 and 21 caninstruct differential deletion in a consistency group unit when thelogical volume 109 that performs the update management in thedifferential management table 202 forms the consistency group.

In this embodiment, the third logical volume has two differentialmanagement tables 202, as described above. If the third storage device30 receives a differential delete request from the firstinformation-processing apparatus 11, it deletes only the differentialmanagement table 202 at the time when the local pair of the firstlogical volume is split the time before last, and begins to record onthe differential management table 202 update information at the timewhen the local pair is newly split.

Transition in General Operation

FIG. 8 illustrates the transition of a data state in each site at thetime T1 to T3 when the first information-processing apparatus 11operates a normal system. Further, STn written in the logical volume 109indicates that the logical volume 109 is consistent with the first mainlogical volume at the time Tn. Further, Δtxy indicates a differentialmanagement table 202 from the time Tx to the time Ty.

First, a state at the time T1 will be described. Assuming that the lastsplit time of a local pair in the first logical volume is T0, a state isillustrated in which the first information-processing apparatus 11splits the local pair of the first logical volume at the time T1. Thestate is as follows: the local pair of the first logical volume is the‘under split’, a remote pair of the first main logical volume and thethird logical volume is the ‘under pair’, and a remote pair of the firstsecondary logical volume and the second logical volume is the ‘undersplit’. The first information-processing apparatus 11 designates thetime T1, which is the same as the split of the first logical volume, andtransmits a delete request from the differential management table 202 tothe third storage device 30. The third storage device 30 having receivedthe request deletes, at the time T1, the differential management table202 recorded at the time when the local pair of the first logical volumeis split the time before last. Further, the second logical volume is inthe state of the first main logical volume at the time T0.

Next, a state at the time T2 will be described. In this state, the firststorage device 10 re-forms a local pair of the first secondary logicalvolume and the second logical volume according to an instruction fromthe first information-processing apparatus 11. The firstinformation-processing apparatus 11 monitors the pairing status of aremote pair of the first secondary logical volume and the second logicalvolume. Then, when the paring status becomes the ‘under pair’, the firstinformation-processing apparatus 11 instructs the first storage device10 to split the remote pair.

Finally, a state at the time T3 will be described. In this state, thefirst information-processing apparatus 11 instructs the first storagedevice 10 to re-form a local pair of the first logical volume. Then, thefirst storage device 10 re-forms the local pair of the first logicalvolume.

As describe above, the transition from the state at the time T1 to thestate at the time T3 repeatedly occurs when the first site is notdamaged. The operation of the first information-processing apparatus 11and the third storage device 30 during this transition will bedescribed.

First, the operation of the first information-processing apparatus 11will be described with reference to FIG. 9. The firstinformation-processing apparatus 11 instructs the first storage device10 to split the local pair of the first logical volume with timedesignation (S911). Here, the split is executed on the local pair in aconsistency group unit. Further, the first information-processingapparatus 11 transmits to the third storage device 30 a differentialdelete request in which the same time as the split is set up as adifferential delete start time (S912). Further, the differential deleterequest is performed on the remote pair in the consistency group, whichis the same as the local pair on which the split is performed. When thefirst information-processing apparatus 11 receives split completionnotice from the first storage device 10 and differential deletecompletion notice from the third storage device 30 (S914) after thesetup time has elapsed (S913), it instructs the first storage device 10to re-form the remote pair of the first secondary logical volume and thesecond logical volume (S915). The first information-processing apparatus11 monitors the pairing status of the remote pair of the first secondarylogical volume and the second logical volume. Then, when the state ofthe pair becomes the ‘under pair’ (S916), the firstinformation-processing apparatus 11 instructs the first storage deviceto split the pair (S917). Subsequently, the first information-processingapparatus 11 instructs the first storage device 10 to re-form the localpair of the first logical volume (S918). The firstinformation-processing apparatus 11 repeatedly executes such a series ofprocesses.

Next, the operation of the third storage device 30 will be describedwith reference to FIGS. 10A and 10B. When the third storage device 30receives a differential delete request from the firstinformation-processing apparatus 11, it sets up a differential deletestart time to a consistency group management table 200 (S1011). Thethird storage device 30 monitors a data write request from the firststorage device 10 during the differential delete start time (S1012).When the third storage device 30 receives the data write request fromthe first storage device 10, it confirms whether data is recorded in thelogical volume 109 in which the differential management table 202 hasnot yet been deleted (S1013). If the data writing is performed in thelogical volume 109 in which the differential management table 202 hasnot yet been deleted, the third storage device 30 compares the time setin the write request with the differential delete start time (S1014).When the time set in the write request is later than the differentialdelete start time, the third storage device 30 first deletes thedifferential management table 202 of a logical volume 109, which is thetarget of the write operation (S1015). The third storage device 30 thenrecords the information regarding the write data, for example, a writeposition on the differential management table 202 (S1016). Further, if adata write request from the first storage device 10 is directed to thelogical volume 109 in which the differential management table 202 hasalready been deleted, the third storage device 30 does not perform thedeletion of the differential management table 202 and records thewritten information on the differential management table 202 (S1017).Further, even though the write is directed to the logical volume 109 inwhich the differential management table 202 is not deleted, the thirdstorage device 30 does not perform the deletion of the differentialmanagement table 202 and records the written information on thedifferential management table 202 when the time set in the write requestis faster than the differential delete start time (S1017).

As such, the third storage device 30 compares the time set up in thewrite request received from the first storage device 10 with thedifferential delete start time set up in the differential delete requestreceived from the first information-processing apparatus 11, andperforms the data writing on the differential management table 202 andthe data deletion from the differential management table 202 in a timesequence. That is, the data writing on the first main logical volume andthe split for the local pair of the first logical volume performed bythe first storage device 10, and the data writing on the differentialmanagement table 202 and the data deletion from the differentialmanagement table 202 performed by the third storage device 30 isperformed in regular order. Accordingly, it is secured that thedifferential management table 202 of the third storage device 30 is awrite history of data written to the first main logical volume after atime when the local pair of the first logical volume is split.

Further, if the third storage device 30 does not receive a write requestfrom the first storage device 10, it deletes the differential managementtable 202 of the logical volume 109 which has not yet been deleted(S1019) after the differential delete start time has been elapsed(S1018). Since the third storage device 30 deletes the differentialmanagement table 202 in a consistency group unit, it confirms whetherthe deletion of the differential management tables 202 for all pairs inthe consistency group is completed (S1020). If the deletion of thedifferential management table 202 for all pairs in the consistency groupis completed, the third storage device 30 deletes the differentialdelete start time from the consistency group management table (S1021)and transmits to the first information-processing apparatus 11 a noticethat the differential management table 202 is completely deleted(S1022). Further, each logical volume 109 has two differentialmanagement tables 202 as described above. Therefore, when the thirdstorage device 30 receives a differential delete request from the firstinformation-processing apparatus 11, it deletes only the differentialmanagement table 202 recorded at the time when the local pair of thefirst logical volume is split the time before last.

[Process when First Site is Damaged]

The operation of making the second logical volume consistent with thefirst main logical volume in order to operate the second site as a mainsystem when the first site is damaged will be discussed. First, thesecond information-processing apparatus 21 instructs the second storagedevice 20 to acquire the paring status of the remote pair of the secondlogical volume and the first secondary logical volume. The secondstorage device 20 having received this instruction refers to the pairmanagement table 202 in the shared memory 104 of the second storagedevice 20 and transmits the contents of the table to the secondinformation-processing apparatus 21.

Next, the second information-processing apparatus 21 instructs thesecond storage device 20 to form a remote pair in which a third logicalvolume is a main logical volume and a second logical volume is asecondary logical volume. Further, the second information-processingapparatus 21 transmits to the second storage device 20 whether theremote pair of the second logical volume and the first secondary logicalvolume was in the ‘resynchronization’ state.

When the second information-processing apparatus 21 instructs the secondstorage device 20 to form a pair of the third logical volume and thesecond logical volume, the second storage device 20 updates the pairmanagement table 202 of the second storage device 20. Further, thesecond storage device 20 transmits to the third storage device 30 thepair formation instruction and the state of the pair of the secondlogical volume and the first secondary logical volume received from thesecond information-processing apparatus 21. When the third storagedevice 30 receives them, it updates the pair management table 202 of thethird storage device 30. Then, the third storage device 30 performs aremote copy from the third logical volume to the second logical volumebased on the state of the pair of the second logical volume and thefirst secondary logical volume, which is received from the secondstorage device 20.

That is, if the state of the remote pair of the second logical volumeand the first secondary logical volume is not the ‘resynchronization’,the third storage device 30 refers to the differential management table202 at the time when the local pair of the first logical volume is splitthe last time. Further, if the state of the remote pair of the secondlogical volume and the first secondary logical volume is the ‘underresynchronization’, the third storage device 30 refers to thedifferential management table 202 at the time when the local pair of thefirst logical volume is split the time before last. The third storagedevice 30 remotely copies only the third logical volume block on thereferred differential management table 202, in which ‘1’ is set up, tothe second logical volume.

FIG. 11A illustrates an example in which the remote pair of the secondlogical volume and the first secondary logical volume is not in the‘resynchronization’ state. Assuming that the time when the local pair ofthe first logical volume is split the time before last is T0 and thetime when it is split in the last stage is T1, FIG. 11A illustrates asituation in which the first site is damaged at the time T3. Since thepairing status of the remote pair of the second logical volume and thefirst secondary logical volume is the ‘under split’, the third storagedevice 30 remotely copies the block recorded on the differentialmanagement table 202 (ΔT13) at the time T1 from the third logical volumeto the second logical volume. Therefore, it is possible to equalize thesecond logical volume, which is ST3, with the first main logical volume.

Next, FIG. 11B illustrates an example in which the remote pair of thesecond logical volume and the first secondary logical volume is in the‘resynchronization’ state. Assuming that the time when the local pair ofthe first logical volume is split the time before last is T0 and thetime when it is split in the last stage is T1, FIG. 11B illustrates asituation in which the first site is damaged at the time T2. Since thestate of the local pair of the second logical volume and the firstsecondary logical volume is the ‘resynchronization’, the third storagedevice 30 remotely copies the block recorded on the differentialmanagement table 202 (ΔT02) at the time T0 from the third logical volumeto the second logical volume. Therefore, it is possible to equalize thesecond logical volume, which is ST2, with the first main logical volume.

In the aforementioned embodiments of the present invention, theembodiments are intended to easy understanding of the present invention,but the present invention is not limited thereto. The present inventionmay be modified without departing the spirit thereof, and includes itsequivalents.

1. A third storage apparatus, coupled to a first storage apparatus whichis coupled to an information processing apparatus and is a copy sourceof both a synchronous remote copy operation to the third storageapparatus and an asynchronous remote copy operation to a second storageapparatus coupled to the first storage apparatus and the third storageapparatus, and wherein the first storage apparatus has a first logicalvolume, and the second storage apparatus has a second logical volume,the third storage apparatus comprising: a plurality of disks configuringa third logical volume including a plurality of data blocks; and acontrol unit, having a memory storing a first bitmap including aplurality of bits and a second bitmap including a plurality of bits, andexecuting reception processing of a remote copy request of thesynchronous remote copy operation from the first storage apparatus, andupdate processing of a data block included in the third logical volumeaccording to the remote copy request, wherein both the plurality of bitsincluded in the first bitmap and the plurality of bits included in thesecond bitmap indicate whether each one of the plurality of data blocksincluded in the third logical volume is updated or not, wherein, after afirst time when the first bitmap is cleared, the control unit sets a bitincluded in the first bitmap, which corresponds to a data block in thethird logical volume targeted by the update processing, as adifferential data between the third logical volume and the secondlogical volume, wherein, after a second time when the second bitmap iscleared, which is different from the first time, the control unit sets abit included in the second bitmap, which corresponds to a data block inthe third logical volume targeted by the update processing, as adifferential data between the third logical volume and the secondlogical volume, and wherein the first and second bitmaps are clearedrepeatedly.
 2. A third storage apparatus according to claim 1, whereinthe control unit sends data blocks corresponding to bits, being set inthe first bitmap or the second bitmap, to the second storage apparatus,in a case of a failure related to the first storage apparatus.
 3. Athird storage apparatus according to claim 1, wherein the third storageapparatus is placed in a third site which is different from a first sitein which the first storage apparatus is placed and a second site inwhich the second storage apparatus is placed.
 4. A method for managingupdates in a third storage apparatus, coupled to a first storageapparatus which is coupled to an information processing apparatus and isa copy source of both a synchronous remote copy operation to the thirdstorage apparatus and an asynchronous remote copy operation to a secondstorage apparatus coupled to the first storage apparatus and the thirdstorage apparatus, wherein the first storage apparatus has a firstlogical volume, and the second storage apparatus has a second logicalvolume, and wherein the third storage apparatus has a third logicalvolume including a plurality of data blocks, a first bitmap including aplurality of bits, and a second bitmap including a plurality of bits,both the plurality of bits included in the first bitmap and theplurality of bits included in the second bitmap indicate whether eachone of the plurality of data blocks included in the third logical volumeis updated or not, the method comprising the steps of: receiving aremote copy request of the synchronous remote copy operation from thefirst storage apparatus; updating a data block included in the thirdlogical volume according to the remote copy request; after a first timewhen the first bitmap is cleared, setting a bit included in the firstbitmap, which corresponds to a data block in the logical volume targetedby the updating step, as a differential data between the third logicalvolume and the second logical volume; and after a second time when thesecond bitmap is cleared, which is different from the first time,setting a bit included in the second bitmap, which corresponds to a datablock in the logical volume targeted by the updating step, as adifferential data between the third logical volume and the secondlogical volume, wherein the first and second bitmaps are clearedrepeatedly.
 5. A method for managing updates according to claim 4,further comprising the step of: sending data blocks corresponding tobits, being set in the first bitmap or the second bitmap, to the secondstorage apparatus, in a case of a failure related to the first storageapparatus.
 6. A method for managing updates according to claim 5,wherein the third storage apparatus is placed in a third site which isdifferent from a first site in which the first storage apparatus isplaced and a second site in which the second storage apparatus isplaced.
 7. A remote copy system comprising: a first storage apparatuscoupled to an information processing apparatus, having a first controlunit and a first logical volume including a plurality of data blocks; asecond storage apparatus coupled to the first storage apparatus, havinga second control unit and a second logical volume including a pluralityof data blocks; and a third storage apparatus coupled to the firststorage apparatus and the second storage apparatus, having a thirdcontrol unit and a third logical volume including a plurality of datablocks, wherein the first storage apparatus is a copy source of both asynchronous remote copy operation to the third storage apparatus and anasynchronous remote copy operation to the second storage apparatus,wherein the third control unit has a memory storing a first bitmapincluding a plurality of bits and a second bitmap including a pluralityof bits, and executing reception processing of a remote copy request ofthe synchronous remote copy operation from the first storage apparatus,and update processing of a data block included in the third logicalvolume according to the remote copy request, wherein both the pluralityof bits included in the first bitmap and the plurality of bits includedin the second bitmap indicate whether each one of the plurality of datablocks included in the third logical volume is updated or not, wherein,after a first time when the first bitmap is cleared, the third controlunit sets a bit included in the first bitmap, which corresponds to adata block in the third logical volume targeted by the updateprocessing, as a differential data between the third logical volume andthe second logical volume, wherein, after a second time when the secondbitmap is cleared, which is different from the first time, the thirdcontrol unit sets a bit included in the second bitmap, which correspondsto a data block in the third logical volume targeted by the updateprocessing, as a differential data between the third logical volume andthe second logical volume, and wherein the first and second bitmaps arecleared repeatedly.
 8. A remote copy system according to claim 7,wherein the third control unit sends data blocks corresponding to bits,being set in the first bitmap or the second bitmap, to the secondstorage apparatus, in a case of a failure related to the first storageapparatus.
 9. A remote copy system according to claim 7, wherein thethird storage apparatus is placed in a third site which is differentfrom a first site in which the first storage apparatus is placed and asecond site in which the second storage apparatus is placed.
 10. Amethod for remote copying in a remote copy system including a firststorage apparatus coupled to an information processing apparatus, asecond storage apparatus coupled to the first storage apparatus, and athird storage apparatus coupled to the first storage apparatus and thesecond storage apparatus, wherein the first storage apparatus is a copysource of both a synchronous remote copy operation to the third storageapparatus and an asynchronous remote copy operation to the secondstorage apparatus, wherein the first storage apparatus has a firstlogical volume including a plurality of data blocks, and the secondstorage apparatus has a second logical volume including a plurality ofdata blocks, and wherein the third storage apparatus has a third logicalvolume including a plurality of data blocks, a first bitmap including aplurality of bits, and a second bitmap including a plurality of bits,both the plurality of bits included in the first bitmap and theplurality of bits included in the second bitmap indicate whether eachone of the plurality of data blocks included in the third logical volumeis updated or not, the method comprising the steps of: receiving aremote copy request of the synchronous remote copy operation from thefirst storage apparatus; updating a data block included in the thirdlogical volume according to the remote copy request; after a first timewhen the first bitmap is cleared, setting a bit included in the firstbitmap, which is corresponds to a data block in the logical volumetargeted by the updating step, as a differential data between the thirdlogical volume and the second logical volume; and after a second timewhen the second bitmap is cleared, which is different from the firsttime, setting a bit included in the second bitmap, which iscorresponding a data block in the logical volume targeted by theupdating step, as a differential data between the third logical volumeand the second logical volume, wherein the first and second bitmaps arecleared repeatedly.
 11. A method according to claim 10, furthercomprising step of: sending step to send data blocks corresponding tobits, being set, in the first bitmap or the second bitmap, to the secondstorage apparatus, in a case of a failure related to the first storageapparatus.
 12. A method according to claim 10, wherein the third storageapparatus is placed in a third site which is different from a first sitein which the first storage apparatus is placed and a second site inwhich the second storage apparatus is placed.